Meta-analysis Research

• Goal: Estimate the true effect size from multiple studies
  
• Steps:
  • define research question (PICOT)
  • collect results from appropriate studies
  • apply statistical model*

Meta-analysis Data

• dataset is \(K\) studies
• each independent study is an observation
• variables:
  • estimated effect size (\(\widehat{\theta}_{k}\))
  • standard error (\(\epsilon_{k}\) or \(SE_{k}\))
  • sample size (\(N_{k}\))

How should we pool effect sizes?

To get our meta-analysis result, we need to take a weighted average of effect sizes…

But how should we weigh the evidence of each study?

Meta-analysis Models

• Fixed effects and random effects models make different assumptions about how data were generated
• Model should approximate the underlying processes

Fixed Effects Assumption

• One true effect size (\(\theta_{\ F }\))
• Deviation from \(\theta_{\ F}\) is due to sampling error (\(\epsilon_{\ k}\))

Inverse-Variance Weighting

• Error (\(\epsilon_{\ k }\)) is inversely proportional to sample size \(N_{k}\).
• Estimates (\(\widehat{\theta}_{k}\)) with small \(\epsilon_{\ k }\) should be closer to true \(\theta\)
• These \(\widehat{\theta}_{k}\) should be given more weight for estimating \(\theta\)

Inverse-Variance Weighting

Study weight \(w_{k}\) is inversely related to the variance (\(SE^2_{k}\))

\[ w_{k}=\frac{1}{(SE)^{2}_{k}} \]

The pooled estimate \(\widehat{\theta}\) is the weighted average:

\[ \widehat{\theta} = \frac{\Sigma_{k=1}^{K} \widehat{\theta}_{k} w_{k}}{\Sigma_{k=1}^{K} w_{k}} \]

FEM Standard Error

To compute the standard error on our averaged result \(\widehat\theta\)

\[ SE_{k} = \frac{1}{\Sigma(w)} \]

FEM Assumptions & Interpretation

• All studies are sampling a completely homogeneous population
• All studies performed in exactly the same manner.
• No between-study heterogeneity
• Too simplistic to account for differences between studies.

Random Effects Assumption

• A universe of possible studies
• Each study has own true effect size (\(\theta_k\))
• \(\theta_{k}\) are randomly drawn from a distribution with mean true effect size \(\mu\) and variance (\(\tau^2\))

Random Effects Model

A study’s effect size (\(\widehat{\theta}_{k}\)) is an estimate of the true study effect size (\(\theta_{k}\)) plus sampling error (\(\epsilon_{k}\)).

\[ \widehat{\theta}_{k}=\theta_{k} +\epsilon_{k} \]

The true effect size of study \(k\) (\(\theta_{k}\)) is drawn from a distribution with mean \(\mu\) and has the new error term \(\zeta_{k}\).

\[ {\theta}_{k}=\mu+\zeta_{k} \]

Random Effects Model

• REM as a hierarchical structure

REM weighting

• we must first establish the variance of the distribution of true effect sizes, \(\tau^2\)
• between-study heterogeneity (\(\tau^2\)) increases the relative weight given to smaller studies (larger SE)

\[ w^{*}_{k}=\frac{1}{SE^{2}_{k} + \tau^2} \]

Then compute the weighted average in the usual fashion.

Estimating \(\tau^2\)

• Many procedures exist, all of which are far too complicated for this talk!
• The DerSimonian and Laird method is one of the oldest and very common

Exchangeability Assumption

• \(\zeta_{k}\) is independent of \(k\)
• there is no way to know what \(\zeta_{k}\) is a priori, ie. without looking at the data
• we say that different studies are assumed to be ‘exchangeable’ because we have no way to know \(\zeta_{k}\)
• This assumption may not be true…

Should we just always use the REM?

• Only choose the FEM if you have clear reason to (rare)

• For most scenarios, use the REM.

A Test for Between-Study Heterogeneity

• does the confidence interval for \(\tau^2\) include zero or not?
• if not, using REM is justified

References & Further Reading

• Cochrane Handbook for Systematic Reviews of Interventions

• Doing Meta-Analysis in R: A Hands-On Guide

• Why I don’t use the term “fixed and random effects”

• Stats, Stat!: Meta-analysis

• An Introduction to Systematic Review and Meta-analysis: Everything you need to know in 59 minutes

Questions?